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System Testing and Early Field 
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(Manuscript received August 14, 1970) 

This article describes the equipment and test procedures used for system 
testing of a large store and forward system. It discusses a novel programmed 
computer load test facility for determining system operational characteristics 
in overload, system traffic handling capacity, and the adequacy of operational 
call register design. It presents early field experience with No. 1 ESS ADF 
controlling a very large nationwide data network for the Long Lines De- 
partment. 

I. INTRODUCTION 

A No. 1 Electronic Switching System Arranged with Data Features 
(No. 1 ESS ADF) was installed in New York City to operate and 
control the Long Lines Department's nationwide Administrative-Data 
Network (ADNet) - 1 The new system was cut into service February 3, 
1969, and connects some 720 Long Lines, Operating Company, and 
Western Electric Company locations to the No. 1 ESS ADF switching 
center through 400 circuits which are terminated by 1,250 four-row 
teletypewriters that use the American Standard Code for Information 
Interchange (ASCII). Connection is also made to computers at two 
Long Lines data processing centers so that computer-generated data 
can be distributed by the network, or field data can be assembled for 
computer center processing. 

Large traffic handling capacity was needed for the ADNet. There- 
fore, a near maximum-sized system has been installed. Two autono- 
mous data scanner-distributors terminate 1,024 full duplex or half 
duplex data lines which operate at speeds up to 150 words per minute. 
Call stores capable of handling 159,744 24-bit words, duplicated, are 
provided. Duplicated program stores of 327,680 44-bit words are pro- 
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vided for storing the 250,000 word program, along with translation and 
parameter data for 1,024 lines. 

Two duplicated disk communities, each capable of storing 2,359,296 
24-bit words, are used for in-transit storage. There are 12 magnetic 
tape units, so that journal filing, permanent filing, and retrieval can 
be handled with six tape units in the on-line pool for retrieval. This 
provides an on-line retrieval capacity of approximately 75,000 
messages. 

The various development testing phases of a message switching 
system of this complexity added up to a sizable undertaking. New 
testing techniques were developed to yield thorough hardware and 
software testing in minimum time. 

II. FUNCTION TESTING 

When testing any large system, it has been conventional to divide 
the work into two main phases, discrete function testing, and system 
operation and maintenance testing. However, it is quite difficult to 
define exactly where function testing for either hardware or software 
ceases and system testing starts. This article uses a somewhat arbi- 
trarily-defined point: that point where all program words had been 
initially debugged (see Fig. 1) and all hardware testing had been 
completed in a fashion similar to a conventional No. 1 ESS sys- 
tem. 2 Hardware testing is not described here. Function and system 
tests are separated to present certain of the testing phases which were 
somewhat uniquely handled in the overall testing program. 

The complete computer program required by the No. 1 ESS ADF 
system contains about 250,000 words. A considerable amount of pro- 
gram testing was required over a period of three years before the sys- 
tem was considered to be operating satisfactorily. The bulk of the 
testing was performed in the test model system laboratory at Bell 
Telephone Laboratories, Naperville, Illinois. The function testing re- 
quired 22 months while system tests started seven months before sys- 
tem cutover and continued for eight months after. The progress in 
program testing, indicated by the amount of program words debugged 
during this period, is shown in Fig. 1. 

During function testing the individual programs were tested as in- 
dependently of other system functions as possible. A test plan was 
devised that would achieve the maximum (i) rate of program integra- 
tion, (u) number of programmers able to use the system effectively, 
and (Hi) number of programmers on day shift. To implement this 
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plan, special utility programs were developed, new laboratory operat- 
ing procedures were established, and a new program test console was 
designed. The console, as shown in Fig. 2, contained some 6,000 lamps 
indicating status in all parts of the system and the information stored 
in the many system registers. It was an expansion of previous ESS 
test console designs with their capabilities, but several important new 
features were added. 2 Facilities were installed for controlling and 
monitoring the frames unique to No. 1 ESS ADF, along with addi- 
tional program and call store address matchers which could be set 
electronically under program control from card input data. The entire 
console was duplicated, but so interconnected that a duplex system 
could be controlled from either half, or the system could be split into 
two simplex systems each independently controlled by one half of the 
console and each half capable of independent operation. This feature 
doubled the machine testing capacity for those programs not requiring 
a duplex system. 

The general concept for the plan of operation was to run the sys- 
tem laboratory as a computation center where programmers would 
work on a single day shift while there would be sufficient console 
operators to complete all the day's batch work in either two or three 
shifts, as required. A programmer could request to be present while 
his work was being run, to observe the system operation and make 
limited changes in his test procedure. All work, however, was under 
control of the console operator. No time was allowed for problem 
study at the machine, on machine time. All problem study was done 
off-line with only completely denned tests being run using machine 
time. 

If a programmer were not present, his work was called a "batch" 
run. If he were present, his work was called a "personal" run. No 
time limit was placed on the length of batch runs, but the longer runs 
were usually performed during night shifts. However, a personal run 
was limited to ten minutes because experience showed that few pro- 
grammers could efficiently use more than ten minutes of time on a 
personal run without requiring at least one-half hour off-line time to 
analyze his results. 

This general plan was adhered to during the period of function test- 
ing and well into the system test period. During the end of the system 
test period before cutover, the request for batch runs dropped off and 
the personal run time was allowed to become longer. After the New 
York system cutover the work became almost entirely personal runs 
with the run time limit extended to 30 minutes. 
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Fig. 3 — Total batch and personal test runs per month. 



At the start of program testing on a test model at Naperville, Illi- 
nois, not all the programs had been loaded. As more programs were 
loaded into the system, the quantity of test runs increased sharply. 
Figure 3 shows the total runs made per month from the start of in- 
tegration until the New York system was cut over in February 1969. 

Figure 4 shows the monthly percentage of the total runs which were 
batch runs during the entire test period. During the debugging of in- 
dividual programs the percentage of batch runs was high, but as the 
testing became more of the system type, the percentage dropped off. 

Figure 5 shows the average length of a test problem. The length of 
each test during the peak program debugging months was kept under 
five minutes. Of this, set-up and read-in of the instruction deck took 
less than one minute, the run itself averaged two minutes, and the 
high-speed printer dump used the remaining time. With the dual input 
console arrangement, the tests were made alternately on each half 
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Fig. 4 — Percent of batch runs to total runs. 

with one half setting up the next run while the other half was making 
a run. This feature greatly helped to keep running time to a minimum. 

Figure 6 shows the monthly rate of program integration which 
reaches a peak of over 18,000 words in a month. Figure 7 shows the 
relationship between words integrated and tests made. The relation- 
ship remained relatively constant throughout most of the integration 
period, indicating that a definite number of test runs must be made by 
a programmer to accomplish a given amount of program integration. 
This seems to be independent of the length of run or the work load 
being performed in the laboratory. 

The relationship between the number of program words integrated 
and hours of machine time is shown in Fig. 8. The overall average was 
37 words integrated per hour of machine time based on the hours when 
the machine was used only for program debugging. If all the hours of 
machine time, that is, overwrite time and maintenance time, are in- 
cluded, the average number of program words integrated per hour 
drops to 30. 
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During the various phases of program testing at the Naperville 
Laboratory, 58,000 overwrite words of program were introduced while 
debugging 134,000 words of program. During the seven months of 
system testing, 22,000 overwrite words were made against the whole 
program of 242,000 words, that is, 9.4 percent of the program was 
changed. During the first eight-month period after system cutover, 
9,700 overwrite words have been put into the system, many because of 
feature additions. Detailed information on the number of overwrites 
made is shown on Figs. 9 and 10. 

111. SYSTEM TESTING 

System testing was conducted primarily on the New York installa- 
tion of the No. 1 ESS ADF system. This installation alone contained 
a full complement of hardware necessary to operate the system at 
full design capacity and to provide a nearly normal environment for 
testing operating procedures. This testing period started seven months 
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before cutover and continued for eight months after. The system tests 
consisted of three distinct but related areas of testing: feature tests, 
network tests, and load tests. 

3.1 Feature Testing 

A detailed test plan was prepared before the start of system testing. 
This plan listed all the tests to be conducted and the system conditions 
and configuration for each test. The tests included all operational and 
administrative features, maintenance test procedures, and system re- 
sponses to abnormal conditions. 

A test bed of stations, not part of the Long Lines network, were 
connected to the New York system for operational feature testing. 
This group of stations consisted of six eight-level ASCII stations and 
nine five-level Baudot stations. There were approximately 500 opera- 
tional tests subdivided into major feature categories as shown in 
Table I. Each test required the preparation of station message tapes; 
and because of the limited number of stations in the test bed, recent 
change tapes had to be prepared for many of the tests to establish 
the proper station options for the given test. Several general purpose 
programs were written to aid in the preparation of these test tapes. 

These tests were systematically run during the early months of the 
system test and were very effective in detecting program bugs. Many 
of the tests were performed while a large system load was being placed 
on the system by the load facilities which are discussed in Section 3.3. 
A subset of these tests were used until cutover to test the introduction 
of new program loads in the New York system. 

3.2 Network Testing 

During the seven months before the cutover the 400 circuits and ap- 
proximately 1250 teletypewriters were installed and tested from manual 
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Table I — Operational Feature Tests 



Heading format 


133 


Addressing 


8 


Precedence 


30 


Privacy 


4 


Multiline hunt group 


15 


Message retrieval 


48 


Poll-delivering criteria 


111 


Undeliverable traffic 


17 


Miscellaneous customer services 


126 



Total 492 



transmission test positions. Five months prior to cutover, the transla- 
tion data for the total Long Lines network was installed in the New 
York system. During this period, one shift of system time was devoted 
to network testing which checked the ability of each station to trans- 
mit and receive traffic, and which verified the translation data. Mes- 
sage tapes were prepared by Bell Laboratories and mailed to station 
attendants for testing the transmission capability of each station. In 
addition, message tapes were prepared for a Long Lines location to 
send to each station for testing that station's receiving capability. 
Since stations continued to be installed throughout this period, special 
network testing programs and procedures were developed to allow the 
translation data to be progressively activated as the stations were in- 
stalled. 

These tests proved to be very effective, and the smooth cutover of 
the network was a direct result of the joint efforts and cooperation of 
Long Lines and Bell Laboratories personnel during this test period. 
The network tests also provided a vehicle for testing the No. 1 ESS 
ADF switching center. Any troubles reported during the tests were 
given the bookkeeping name of failure reports, and corrections were 
introduced into the program. Figure 11 shows the number of failure 
reports issued during system testing. Prior to cutover, there were 690 
reports per month. While the network tests were responsible for a large 
number of these failure reports, load tests and feature tests which 
were conducted concurrently and the continued testing on the Naper- 
ville test model contributed significantly to these results. After cut- 
over the rate dropped to an average of 135 reports per month for the 
next five months and then dropped to near zero. These failure reports 
were divided into three classes : 

Class A, serious service-affecting troubles which were corrected im- 
mediately (less than five percent) . 
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Fig. 11 — Total number of failure reports. 

Class B, refinements to service or maintenance which were corrected 
more routinely (over 80 percent) . 

Class C, corrected only if simple, otherwise deferred for later pro- 
gram versions. 

3.3 Load Testing 

A switching system cannot be considered to be fully tested until it 
is stressed to the limits of its traffic handling capability. However, the 
large traffic capacity of the No. 1 ESS ADF switching center makes 
live load testing difficult because of the problem of generating the large 
amounts of controlled traffic required to stress the system to capacity. 
To fully load the system with traffic on each line is impractical be- 
cause the total cost of hardware becomes prohibitively great. Even if 
one could afford the hardware, it is virtually impossible to administer 
the generation of traffic to produce a controlled load which is re- 
producible. Therefore, load testing of the No. 1 ESS ADF system was 
accomplished through the use of two load boxes which were developed 
for this purpose. The test configuration for the load boxes, a five-level 
load box and a computer-controlled load box, is shown in Fig. 12. 

3.3.1 Five-Level Load Box 

The five-level load box was designed as a means of testing the data 
scanner distributor units under full load. This device consisted of ten 
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paper tape readers each of which could be connected to as many as 52 
ports on the data scanner distributor units. The load box thus simu- 
lated 520 five-level full duplex stations. The five-level full duplex 
signalling sequence is such that after the initial handshaking with the 
station to establish the connection, a multimessage transmission can 
be continued indefinitely since no further signaling dialogue between 
the station and the switching center is required. The control teletype- 
writer shown in Fig. 12 was used to handle the initial signaling se- 
quences for each of the paper tape readers. Once the connection was 
established, the control teletypewriter was then available for use with 
another tape reader. While this load box arrangement was designed 
primarily for testing the data scanner distributor units, it had some 
characteristics which produced useful program tests. Since a paper 
tape reader was connected to 52 lines, the traffic presented to these 
lines was not random, that is, all actions such as start and end of mes- 
sage occurred simultaneously. This in turn produced stresses on par- 
ticular program functions as each action occurred. This load box was 
an effective testing device in the early stages of system testing, but 
had to be removed when the network tests started to make way for 
the Long Lines network. 
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Fig. 12 — Load box facilities used in testing the No. 1 ESS ADF system. 
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3.3.2 Computer-Controlled Load Box 

A computer- controlled load test facility was developed for the No. 
1 ESS ADF project. This load box consisted of a small computer, the 
Honeywell DDP-516, and hardware connecting it to the ADF system 
in such a way as to simulate a data scanner distributor unit with 512 
lines and two stations per line. The program written for the DDP-516 
computer provides a large degree of flexibility in controlling the traffic 
parameters used to specify the type of traffic load presented to the 
system. 

These parameters are: (i) the number of active originating stations, 
(ii) the level of presented load called message presentation rate, (ra) 
the multiple address factor or number of addresses per message, and 
(iv) the text length of the message. The latter three parameters can 
be provided with constant, normal, or exponential distribution. Traf- 
fic statistics collected by the load box are periodically printed and 
have proven to be valuable in evaluating the No. 1 ESS ADF system 
operation. 

The DDP-516 computer was so programmed that steady state or 
dynamic load testing could be performed. In a steady state test, the 
traffic parameters remain fixed during the test. This allows the system 
response to a steady load to be observed. In a dynamic test, the basic 
traffic parameters for the load box are varied with time according to 
a pattern. This allows the system response to varying traffic conditions 
to be observed. 

The first use of the load box was to supply a background load while 
feature tests were performed on the system. Subjecting the No. 1 ESS 
ADF to traffic loads while making discrete functional tests of individ- 
ual programs exposed program faults which otherwise would have re- 
mained undetected. Many of the program faults consisted of subtle 
interactions within the program which had a very low probability of 
occurring under light load. Such situations could be created by large 
volumns of traffic which would never have been produced manually. 
The ability to control the traffic and reproduce the tests made it pos- 
sible to examine a given fault repeatedly. 

3.3.3 System Capacity Tests 

To build a foundation for future traffic engineering of the Long 
Lines network, it was necessary to make dynamic system load tests 
to gather information concerning the capacity limits of the system. 
In addition, system performance under heavy loads and overloads 
needed to be demonstrated. Since the system is now only partially 
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loaded from the Long Lines network, the use of the computer-con- 
trolled load box was the only means of obtaining the desired test re- 
sults. 

The large number of system problems encountered before cutover 
and in the early months after cutover, and indicated in Fig. 11, made 
it impossible to perform meaningful capacity tests. However, these 
problems were gradually fixed and by September 1969, all the known 
software and hardware problems had been eliminated. A week-long 
test of system capacity and overload performance was conducted dur- 
ing September 1969. 

The No. 1 ESS ADF system uses the time required by the program 
to complete the main program cycle (called the E-to-E cycle time) 
as a measure of the load on the system and as a means of initiating 
load control procedures. To aid in collecting data on real-time capacity 
tests of the system, additional measuring equipment was provided 
with the load box. This consisted of an electronic counter which was 
triggered upon the completion of each main program cycle of the No. 
1 ESS ADF processor. The counter was started on odd pulses and 
stopped on even pulses. The results of the counter were read by the 
load box, thus the load box collected measurements of every other 
main program cycle time. These data were printed periodically as a 
part of the load box traffic statistics with the maximum, the minimum, 
and the average main program cycle time for the preceding period. 

The load box also predicted the capacity based on measurements 
taken in the preceding measurement period. The prediction was an es- 
timate of 90 percent of the limiting capacity, that is, 90 percent of the 
capacity at which the E-to-E cycle time would be infinite. The 90 per- 
cent level was chosen as the best guess of the level of capacity which 
would be achieved with the current load control parameters. The pre- 
diction was based on this equation which was derived from theoretical 
analysis of the system capacity. 

where: 

C(0.9) = 90 percent of the limiting capacity in characters per 
second. 
E A = Average E-to-E cycle time in ms. 
E N = No load E-to-E cycle time in ms. 
T — Average capacity in characters per second during the 
measuring interval. 
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For some values of traffic characteristics, call store size and not 
real time, limits the system capacity; thus a prediction method was 
necessary to determine real-time capacity. Also, the load box predic- 
tion was useful during load box runs to establish changes in the input 
load box parameters to establish the desired load level. The data re- 
quired for the calculation of equation (1) was measured by the load 
box during each test interval with the exception of E N , the no load 
E-to-E cycle time. This quantity which was an input parameter to 
the load box was a constant for the purpose of the prediction calcula- 
tion. The value used was the average E-to-E cycle time observed dur- 
ing long periods of operation with no load on the system and was about 
50 ms. The variation in E N during the test runs results in a prediction 
error in the calculation of equation (1). The value of the error de- 
creases as the load increases and has been observed to be less than 
3 percent for values of the capacity in excess of 50 percent of the 
real-time capacity. 

The real-time capacity of the ADF system is a function of multiple 
address factor, message length, and statistical distribution of message 
length. Past estimates of capacity based on calculations rather than 
on measurements have indicated that the multiple address factor 
(Af), the number of addresses per message, has a minor effect on 
capacity. To validate this, tests were made with M = 2 and M = 3 
for the same average message length. The results indicated only a 5 
percent increase in capacity for the higher multiple address factor. 
This confirmed the original assumption so that the major effort was 
directed at determining real-time capacity as a function of message 
length and its statistical distribution, and all other tests were made 
with a multiple address factor of 2. 

Two test runs, fixed message length and exponential distribution of 
message length, were made for four average message lengths: 100, 
500, 900, and 1,300 characters. The call store rather than real time 
was found to be the limiting factor on capacity for all average mes- 
sage lengths greater than about 300 characters. The call stores could 
support more capacity if the data rate on the lines were increased. For 
this reason, the load box was programmed to produce a line rate of 
150 words per minute rather than the normal 100 words per minute 
for the tests made at the three longer message lengths. A load was ap- 
plied to the system and allowed to stabilize. Recordings of capacity, 
predicted capacity, and minimum, maximum, and average E-to-E 
cycle were taken. The load was increased and the process repeated 
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AVERAGE MESSAGE LENGTH ( HEADING AND TEXT ), x 100 CHARACTERS 

Fig. 13 — Real time average and limits of capacity measurements for (•) fixed 
message length and for (x) exponential distribution of message length. The 
multiple address factor is 2.0. 



until the real-time load control level was reached or the call store 
limited further increased in the load. 

The predicted capacity, 90 percent of the limiting capacity, as a 
function of the average message length for the two types of message 
distributions, is given in Fig. 13. The prediction error should be low 
since capacity levels of 75 percent or higher were reached during each 
test run. 

The tests were analyzed to determine whether the ADF system 
could operate at 90 percent capacity and, even further, whether it 
should. To understand this analysis, it is necessary to consider the 
functional relationship between the average E-to-E cycle time and 
the capacity, and how the capacity is limited by the load control pa- 
rameters. The functional relationship between the average E-to-E 
cycle time and the capacity is a family of curves dependent on mes- 
sage length which tends to make the present considerations quite 
complex. To simplify the analysis, only the functional relationship be- 
tween the average E-to-E cycle time and normalized load were con- 
sidered, where normalized load is defined as the ratio of the amount 
of data that pass through the machine to the limiting capacity. This 
results in a single curve independent of message length. This relation- 
ship is given in Fig. 14 based on the load box data. At the load con- 
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Fig. 14 — Average E-to-E cycle time versus normalized load (ratio of data 
passing through the computer to its limiting capacity). 

trol level, currently set in parameters as 400 ms, the capacity is kept 
by the load control procedures to approximately 88 percent of the 
limiting capacity, not quite 90 percent. 

To determine if the load control level is set properly, the effects of 
peak rather than average E-to-E cycle time must be considered as 
two other actions are initiated as a result of parameter value com- 
parisons with E-to-E cycle time, (i) If the E-to-E cycle time exceeds 
3.24 seconds, a real-time overload will be declared and certain opera- 
tional tasks will be suspended, (n) If the E-to-E cycle time exceeds 
4.32 seconds, phases of emergency action will be initiated. 

During tests with the capacity at 85 to 90 percent, peak values of 
E-to-E cycle time went very high — about eight times the average 
E-to-E cycle time. However, the number of occurrences of real-time 
overload was small and it appeared that the recovery was rapid, that 
is, the real-time overload ended in the E-to-E cycle following the one 
in which it began. The overload appeared to have no effect on service; 
thus the current value of 400 ms for the load control level appears to 
be quite reasonable. With this control level, the system will operate 
at about 88 percent of the limiting capacity. The capacity curve of 
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Fig. 13 for the exponential distribution of message length appears to 
be a conservative representation of the real-time capacity of the ADF 
system for the Long Lines traffic characteristics. 

To further confirm this conclusion, a load test of the No. 1 ESS 
ADF system was made during a normal traffic day. Throughout the 
test period, 8:00 a.m. to 6:00 p.m., the traffic data produced by the 
system was monitored. The load box parameters were adjusted to 
maintain a system traffic intensity that was three times that produced 
by the Long Lines network alone while maintaining the traffic profile, 
that is, the length of busy hours was unchanged. During the course of 
this demonstration, all the traffic was handled without an overload, 
although some of the call store facilities were in very heavy use. To- 
ward the middle of the afternoon, it was decided that an overload 
should be forced on the system. Therefore, the load box traffic was 
sharply increased to force a call store overload. Load control was 
automatically called in and, after the overload was removed, the sys- 
tem recovered from the overload without incident or loss of messages. 
This test was a demonstration to establish performance credibility 
and was not used to collect capacity data. 

3.3.4 Queueing Tests 

A series of tests was designed to show that the queueing logic, im- 
bedded throughout the operational program, operated properly for 
each and every call store facility. A method was devised, using a small 
number of program store overwrites, to artificially reduce the number 
of items assigned to any given call store facility. For the call store 
facility under test, the number of items available could be controlled 
and changed by setting the appropriate value in a call store location. 
With the load box providing a high level of load on the system, but 
with the load controlled to a level that produced no facility queueing, 
the number of items for the facility under test was reduced to where 
virtually continuous queueing existed. The system was operated for 
approximately one hour in this state to exercise as many points as 
possible in the program which involved queueing for the facility being 
tested. Audits were run periodically to detect errors, and the perform- 
ance of the system was monitored. It should be obvious that there 
is no assurance that all points in the program which involve queueing 
will be tested by this process ; however, those with a reasonable prob- 
ability of occurrence will have been thoroughly exercised. All call 
store facilities were tested through this process. 



2994 THE BELL SYSTEM TECHNICAL JOURNAL, DECEMBER 1970 

In addition to the call store facility queueing tests, a test of the 
performance of the system during a message store overload was made. 
Load control procedures were initiated which allowed current inputs 
to continue until they were complete, but restricted the acceptance of 
any further input traffic until the output process could make storage 
space available. 

3.3.5 Call Store Tests 

Capacity of a store and forward system is generally defined in 
terms of the system's thruput capability, that is, the number of 
characters transmitted and received by the switching machine per 
unit of time. In line-switched electronic switching systems, capacity 
is primarily limited by the real-time capability of the central proces- 
sor. However, in store and forward systems, the requirements for 
storage are much greater. The system is committed to accepting origi- 
nating traffic without regard to whether the traffic can be delivered to 
the terminators. Large amounts of storage are required to hold or 
queue the traffic for delivery to the terminators in addition to the 
storage required for the bookkeeping associated with transmission to 
and from the switching machine. In the No. 1 ESS ADF system, the 
call store capacity, as well as the real time of the central processor, 
limit the system capacity under certain traffic characteristics. Storage 
space is assigned by two basic considerations : that which is associated 
with transmission — the greater the storage, the higher the thruput 
that can be obtained — and that which is associated with holding traf- 
fic in the system — the greater the storage, the longer the traffic may 
be held in the system — which, in turn, means that transmission facili- 
ties can be used more efficiently. Both of these functions are vying 
for the same pool of finite storage. 

The current call store allocation for the New York system was 
based on a traffic survey made by the Long Lines Traffic Department. 
The recommended assignment of call store for all traffic-dependent 
areas based on this design load is shown in Table II. The five major 
items — input processing registers, output processing register, assembly- 
disassembly blocks, message processing blocks, and message queue 
registers — require 88 percent of the assignable area with message proc- 
essing blocks requiring more storage than all other items. For this 
reason, the major traffic engineering effort was placed on these five 
items in the development of assignment rules. 

To test the adequacy of the call store assignments, several load box 
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Table II — Present Call Store Assignment For Traffic Depend- 
ent Facilities 





Words 










per 


Number 


Total Num- 




Facility 


Register 


Assigned 


ber of Words 


Percent 


Assembly disassembly blocks 


32 


544 


17,408 


16.2 


Input processing registers 


16 


150 


2,400 


2.2 


Output processing registers 


19 


331 


6,289 


5.8 


Message processing blocks 


32 


1645 


52,640 


48.7 


Message queue registers 


3 


5500 


16,500 


15.2 


Other items 






2,724 


2.5 


Unassigned area 






10,038 


9.4 


Total 


107,999 


100 



runs were made varying the traffic intensity of the presented load. 
During these runs, the quarter-hour traffic printouts provided by the 
ADF system were used for capacity measurements and for average 
usage of each call store facility. Peak values for the call store usage 
counts were obtained by reading the traffic counters inside the ADF 
system about once per minute. 

The peak and average usage of each facility as a function of traffic 
load was analyzed. As a result, the present procedures for engineering 
call store appear to be quite valid, with some minor changes. Making 
use of these changes in the traffic engineering procedures, Fig. 15 il- 
lustrates an approximate relationship between the number of call store 
words required per input line and the average input load per line. The 
number of call store words include the requirements for input trans- 
mission (input processing registers, assembly-dissassembly blocks, and 
message processing blocks) , output transmission (output processing 
registers, assembly-disassembly blocks, and message processing blocks) , 
and holding traffic on the message queue (message queue registers). 
This illustrates the design choice between call store facilities for input 
and output transmission and facilities for delaying traffic. As the input 
load increases, a breaking point is reached where the load on each 
output line is one erlang. At this point there is a choice of adding 
more output lines and associated call store output transmission facili- 
ties or adding only message queue registers to hold the traffic for fu- 
ture delivery. The latter is less expensive if one can put up with the 
increased service delays. 

Some knowledge of the future traffic requirements is needed to make 
recommendations concerning future traffic engineering and call store 



2996 



THE BELL SYSTEM TECHNICAL JOURNAL, DECEMBER 1970 



assignment of the ADF system. Call store must be put to the most ef- 
fective use to reduce transmission plant cost while meeting the traffic 
needs. The maximum number of call stores were provided with the 
New York system. It is, therefore, assumed that the network load 
will increase as a result of the addition of stations to the present plant 
in preference to the addition of large quantities of transmission facili- 
ties. In the process, the load on individual lines should be balanced. 

Because future traffic requirements are unknown, Table III proposes 
an allocation of call store in which a traffic load of about 3.5 times 
the present Long Lines load can be handled. If the Long Lines traffic 
forecast indicates that capacity levels are needed beyond that which 
can be achieved with the call store allocation of Table II, additional 
transmission plant cost will be incurred. By converting all the 100 
words per minute lines to 150 words per minute, the amount of data 
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Fig. 15 — Approximate call store requirements per input line as a function of 
load per input line for the major traffic dependent registers. S = Service delay 
in minutes. R = Ratio of output to input lines. Output message length: 1200 
characters. "Busy hour": 1 hour followed by a low traffic period for emptying 
message queues. Multiple address factor: 3.0. 
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Table III — Proposed Call Store Allocation For Traffic Depend- 
ent Facilities 



Facilities 


Words 

per 

Register 


Number 
Assigned 


Total 
Words 


Percent 


Assembly-disassembly blocks 
Input processing registers 
Output processing registers 
Message processing blocks 
Message queue registers 
Other items 


32 
16 
19 
32 
3 


711 

182 

415 

1,333 

10,006 


22,752 

2,912 

7,885 

42,656 

30,018 

1,776 


21.1 

2.7 

7.3 

39.5 

27.8 

1.6 




107,999 


100 



passing through the computer would approach the system's real-time 
capacity, and a traffic load of about five times the present Long Lines 
load could be handled. The system capacity limits are summarized in 
Fig. 16. 



IV. field experience 



By May of 1970, the No. 1 ESS ADF switching system at 811 Tenth 
Avenue, New York, had been in continuous operation for 15 months. 
During that time, hardware and software changes were introduced to 
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correct troubles encountered, new service features were added, and a 
final generic program was installed without interrupting service. 

Although certain operational difficulties were encountered during 
the early months, the system has performed satisfactorily. The net- 
work is now being used to handle all Long Lines administrative mes- 
sages, commercial service orders, traffic service orders, service results, 
payroll, plant circuit orders and expense analysis reports. As shown in 
Fig. 17, over 20,000 messages (originated plus terminated) were 
handled daily in the first month of operation. As new projects were 
added to the network, the daily traffic grew to 35,000 messages averag- 
ing 1,200 characters each. Additional traffic will be added during the 
next year to increase the load to 120,000 messages daily. 

4.1 Service Experience 

4.1.1 Grade of Service 

The quality of service of a store-and-forward system may be mea- 
sured by the: (i) delivery time of messages, (u) number of lost or mis- 
directed messages, and (Hi) number of station troubles per day. Be- 
cause of the nature of store-and-forward service, no one criterion tells 
the whole story. 

The pick up and delivery time of messages depends on how heavily 
a multistation line is loaded and, therefore, is a function of engineer- 
ing rather than switching center service. On lightly loaded lines, the 
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Fig. 17— Administrative-data network average daily traffic, with an average 
message length of 1200 characters. 
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pick up is immediate and the delivery to any station on the admin- 
istrative-data network is less than one minute. On heavily loaded lines, 
pick up delays may be as long as 20 minutes and delivery may be 
delayed varying amounts depending on the length of the queue for 
the terminating station. Customer reaction to service has been excel- 
lent, but some additional lines have been added in some locations to 
reduce overload on key message center lines. 

During the first week of operation, some messages were lost during 
a duplex disk failure. Since that time over 10,000,000 messsages have 
been handled without loss or misdirection of messages accepted by 
the system. About one message in 5,000 is undeliverable and the origi- 
nator is asked to send the message again. 

The number of station and transmission troubles is about one-fifth 
the monthly Bell System average for conventional teletypewriters in 
the field. This improved performance can be attributed to the teletype- 
writer electronic controller, new data sets, and continuous monitoring 
of the network by the No. 1 ESS ADF switching center. 

4.1.2 Message Retrieval 

The number of retrieval requests have been averaging about 250 
per day or about 0.7 percent of daily traffic- These retrievals are 
needed by the user because of station problems, loss by local at- 
tendant, delivery to an additional terminator, or desire for a second 
copy. The breakdown for each of the above categories is not known 
in detail, but most retrievals result from station problems. The amount 
of retrieval traffic is below the level predicted and the retrieval service 
given has proven satisfactory to the user. 

4.1.3 Network Management 

The network management center described in Ref. 3 has proven to 
be very effective in managing a network as large as the Administrative- 
Data Network. On many occasions when traffic bottlenecks occurred 
because of transmission or terminal outages, network management per- 
sonnel were able to pinpoint these bottlenecks and reroute the traffic 
until the line or station troubles were cleared. In addition to the spe- 
cific network management features, the memory capacity of the sys- 
tem also is helpful in handling large backlogs of traffic. For example, 
the system provides for an interconnection with off-line commercial 
computers which process payroll data and other projects. Because 
these commercial computers are occasionally out of order, messages 
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destined for the data processing center cannot be delivered and traffic 
in excess of 1,000 messages has been stored and queued for delivery in 
No. 1 ESS ADF for a day or more. 

4.1.4 Network Maintenance 

The normal maintenance procedure for data transmission and sta- 
tion plant is to test and repair facilities in the light of customer 
trouble tickets. To identify trouble before a customer reports it, No. 1 
ESS ADF has been designed to locate trouble within seconds and re- 
port the trouble automatically to the maintenance center. This is pos- 
sible because the switching system is programmed to poll every sta- 
tion not originating or terminating traffic every two seconds. By 
analyzing the results of polling and other "handshaking" signals, it is 
possible to determine station problems such as power failure, teletype- 
writer out of paper, jammed paper, or general station failure. Open, 
shorted, or noisy lines are also detected and reported to the mainte- 
nance center. 

These automatic trouble reports now supplement customer trouble 
reports and, in many cases, malfunctions are being fixed before the 
customer is aware of a problem. The average time to repair admin- 
istrative-data network data line and station troubles is below the Bell 
System average for similar equipment. 

4.1.5 Addition of New Features 

As a result of early operational experience, several new features 
seemed desirable to improve system performance. For example, im- 
provements were made to identify the specific error in a heading 
format of a rejected message so that the originator could more easily 
make corrections. Provisions were also made for automatic control of 
a tape punch or other auxiliary equipment at a station by using ad- 
dress mnemonics. Features also were added to improve the procedure 
for introducing translation changes for new lines and stations. 

The new or improved features were introduced temporarily with- 
out service interruption and were all included in the final generic 
program. 

4.2 Operational Problems 

4.2.1 System Down Time 

The No. 1 ESS ADF switching center and the network it controls 
are designed for continuous operation without loss of messages or de- 
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lays in message delivery. As described in Itefs. 3 and 4, the programs 
were designed so that auditing programs could monitor and correct 
any call store words which were mutilated by software or hardware 
malfunctions. In addition, special programs called "emergency action 
programs" were provided to take care of excessively mutilated data 
which could not be handled by audits, or failures of system operation 
which could not be cleared by simple maintenance. These programs 
restart certain registers and memory locations and make it possible for 
the system to recover and continue processing. 

Such emergency action programs are divided into four successive 
phases, each automatically called in if the situation is not corrected 
by the preceding one. The first three are relatively short and do not 
interfere with data processing. Phase 4 lasts about 40 seconds and 
does interrupt processing. However, this does not significantly affect 
service since originations are queued for input and output and the 
delay is not noticeable to the customer. Messages already accepted by 
the system are not mutilated, but messages being originated while a 
phase 4 program is in progress are aborted and the originator is auto- 
matically advised to send the message again. 

These defensive tactics proved to be very effective in reducing sys- 
tem down time during the early months after cutover as they were 
able to maintain satisfactory service even with hardware and soft- 
ware troubles. Figure 18 shows the system down time since cutover. 
The relatively high down time shown during February and March 
resulted from program and hardware troubles not detected until after 
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Fig. 18 — System downtime in minutes. 
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a continuous live load had been placed on the system. These troubles 
were rapidly corrected. Total down time per month then dropped 
dramatically and has varied from 1 to 5 minutes per month. During 
January 1970, when the final generic program was introduced, several 
phase 4 emergency programs were required before the system would 
run satisfactorily on the new generic program. These emergencies ac- 
cumulated 10 minutes of down time during the midnight hours. 

4.2.2 Types of System Problems 

Analysis of trouble reports show that software accounted for many 
of the early troubles. Program troubles were encountered in the tape 
retrieval system when it was under heavy retrieval load. Unfor- 
tunately, the load test facilities used before cutover were not capable 
of testing a heavy message retrieval load. Several other problems not 
sensitive to load, but to improper system operation by the user became 
evident. For example, one user tried to send a 905-multiple address 
message. The design limit is 379 addresses, with a check to reject mes- 
sages asking for a greater number. However, the program was in error 
and did not detect the illegal request, so the machine switched auto- 
matically to emergency action programs. 

Some maintenance programs were too sensitive. For example, thresh- 
olds for allowable error rates were set too low and, in some cases, the 
maintenance strategy of system reaction to malfunction had to be 
revised. 

After the initial software troubles were corrected, the remaining 
troubles generally resulted from either hardware failures or improper 
procedures by maintenance personnel. Circuit pack failures were few 
and in line with previous data, as reported in Refs. 5 and 6. 

Two analog units in the system, the tape transport and the disk 
file, required the most attention. The tape transport is sensitive to dirt, 
tape wear, and transport adjustments. Since 12 tape units are avail- 
able, the problems were not critical to system operation, but they did 
require considerable maintenance. New alignment procedures, cleaner 
rooms, and close inspection of magnetic tape quality has reduced the 
troubles. 

The disk file had failures in head diodes. An improved diode is 
now available, which is gradually being put into service. The disk file 
also had several "head crashes;" that is, the head touched the rotating 
disk, scratching off the magnetic coating. Such head-touching prob- 
lems can most likely be attributed to dirt. New routines have been set 
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up for cleaning the sealed disk units periodically and replacing the 
0.2 micron dirt filter more frequently. The long term solution appears 
to be the development of a closed self-purging system so that air is 
recirculated and dirt is not allowed to accumulate. 

v. CONCLUSION 

The development of No. 1 ESS ADF has resulted in the design of 
several new types of hardware and the creation of some new testing 
techniques. The introduction of this system puts into Bell System 
operation for the first time disk files, acoustic delay lines, magnetic 
tape retrieval systems, and an electronic autonomous scanner-distribu- 
tor for data lines. Newly introduced are: batch program debugging on 
an ESS machine, dual position program test console with matchers 
that can be set electronically, program controlled computer load test- 
ing, real-time printout of monitor dump facilities, and a complete set 
of improved utility programs to facilitate program compiling, loading, 
debugging, and insertion of program changes. 

The operation of the new system at 811 Tenth Avenue, New York, 
since cutover, has been very satisfactory. Although the system is not 
being loaded to handle maximum traffic now, a live load test was run 
to demonstrate very large traffic handling capability and the ability 
to operate in real-time overload without mutilating, aborting, or losing 
messages. The measured traffic handling capacity of this system is a 
function of message length. For example, the system can handle 19,000 
messages per hour for 1,200 character messages or 33,000 messages per 
hour for 400 character messages. 

The system has proven to be reliable and during the last 15 months 
has handled 10,000,000 messages without loss. The monthly system 
down time is averaging about two minutes per month with excellent 
prospects for further reduction as more experience is gained with the 
system. 

The field experience gained since cutover confirms the effectiveness 
of the service features, the adequacy of system traffic capacity and 
the reliability of system operation. 
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